Algorithmic Differentiation of Numerical Methods: Second-Order Tangent and Adjoint Solvers for Systems of Parametrized Nonlinear Equations

Forward and reverse modes of algorithmic differentiation (AD) transform implementations of multivariate vector functions F : IR → IR as computer programs into tangent and adjoint code, respectively. The reapplication of the same ideas yields higher derivative code. In particular, second derivatives play an important role in nonlinear programming. Second-order methods based on Newton’s algorithm promise faster convergence in the neighbourhood of the minimum by taking into account second derivative information. The adjoint mode is of particular interest in large-scale gradient-based nonlinear optimization due to the independence of its computational cost on the number of free variables. Solvers for parametrized systems of n equations embedded into the evaluation of the objective function for a (without loss of generality) unconstrained nonlinear optimization problem require the Hessian of the objective with respect to the free variables implying the need for second derivatives of the nonlinear solver. The local computational overhead as well as the additional memory requirement for the computation of second-order tangents or second-order adjoints of the solution vector with respect to parameters by a fully algorithmic method (derived by AD) can quickly become prohibitive for large values of n. Both can be reduced significantly by the second-order symbolic approach to differentiation of the underlying numerical method to be discussed in this paper.